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SOME  LEAST -SQUARES  TRANSFORMATIONS  OF  REGRESSION 
ESTIMATORS  OF  ORTHOGONAL  FACTORS 

Emil  F.  Heermann 

U.  S.  Army  Personnel  Research  Office 

This  paper  provides  a brief  introduction  to  the  factor  analysis  model 
used  by  psychologists  and  presents  some  methods  for  transforming  re- 
gression estimators  of  orthogonal  factors  to  a more  useful  form. 

I.  THE  FACTOR  ANALYSIS  MODEL.  Factor  analysis  is  a mathe- 
matical method  for  representing  n correlated  measures  obtained  on  N 
individuals  by  means  of  a linear  combination  of  hypothetical  measures, 
called  factors.  Psychologists  draw  a distinction  between  common  factors, 
which  are  related  to  two  or  more  measures,  and  specific  factors  which 
are  related  to  only  one  of  the  original  measures.  In  general,  then,  the 
linear  model  postulates  n specific  factors  and  r common  factors  where 
r is  less  than  the  number  of  measures  n.  The  sampling  theory  for 
such  a model  is  not  yet  thoroughly  understood  and  this  paper  will  con- 
fine itself  to  algebraic  rather  than  statistical  considerations. 

The  linear  model  of  factor  analysis  can  be  most  conveniently  pre- 
sented in  the  form  of  matrix  equations: 

(1)  Z = ZCF'  + ZgS'  , where 

Z = N x n matrix  of  observed  measures  (zero 
mean,  unit  variances) 

Zc=  N x r matrix  of  common  factor  scores 

F1  = r x N matrix  of  common  factor  weights 

Zg  = N x n matrix  of  specific  factor  scores 

S'  = n x n diagonal  matrix  of  specific  factor 
weights 


Terms  on  the  right  side  of  the  equation  are  all  unknown.  In  (1)  it  is 
assumed  that: 
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These  assumptions  allow  us  to  find  F'  and  S'  from  the  observed 
measures,  but  before  we  examine  this  procedure  it  is  important  to  indi- 
cate a rather  serious  difficulty  with  our  linear  model.  When  we  postu- 
late n + r factors  to  explain  n measures  this  invariably  means  that 
there  exist  multiple  solutions  for  an  individual's  factor  scores.  Solutions 
exist  because  the  linear  equations  are  consistent  but  the  rank  of  the  co- 
efficient matrix  does  not  allow  unique  solutions.  To  get  around  this 
difficulty  psychologists  have  utilized  least- squares  procedures  for  esti- 
mating the  factor  scores  from  the  observed  measures,  and  we  shall 
examine  the  defects  of  these  estimators  at  a later  point. 

We  now  return  to  the  problem  of  finding  F'  and  S'  from  the  observed 
data.  Because  of  (2),  (3)  and  (4)  we  may  write: 


R = — Z'Z  = FF'  + SS',  or 
N 


R - SS'  = FF' 


Since  SS'  is  a diagonal  matrix,  we  begin  by  estimating  values  for  SS'  and 
then  obtain  the  eigenvectors  and  eigenvalues  of 


R - SS'  = AKA',  where 


A = n x r matrix  of  eigenvectors 


K = r x r matrix  of  eigenvalues 
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Now,  F = AK  ' , and  the  sums  of  squares  of  the  rows  of  F are  used 

to  obtain  new  values  of  SS1,  and  the  process  is  continued  until  R - SS1 
is  fit  with  the  minimum  rank  F.  This  whole  process  may  be  thought  of 
as  finding  a set  of  values  for  SS1,  such  that  R - SS1  is  of  minimum  rank. 

It  will  be  noted  that  F is  arbitrary  up  to  an  orthogonal  transformation 
and  it  is  necessary  to  postulate  some  method  for  finding  a psychologically 
meaningful  F.  Psychologists  follow  L.  L.  Thur stone  here  and  attempt 
to  transform  the  arbitrary  F,  so  that  it  approximates  "simple  structure", 
i.  e.  , has  a maximal  number  of  zero  or  near-zero  entries.  To^accom- 
plish  this  we  find  some  orthogonal  transformation,  • such  that 


where 


FA  = fr 

X X =1,  and 


Fr  approximates  "simple  structure".  A considerable  number  of  computer 
programs  exist  which  determine  by  analytical  means  the  best  transfor- 
mation X • 

II.  ESTIMATION  OF  FACTOR  SCORES.  Assuming  now  that  F 
is  adequately  fixed,  our  problem  is  then  to  find  the  values  of  Zc  and 

Zg  that  will  satisfy  the  linear  model  specified  in  (1).  Since  an  infinite 
set  of  such  factor  scores  exist,  psychologists  commonly  turn  to  a re- 
gression method  for  estimating  Zc  for  a fixed  Zg.  We  are  primarily 

interested  here  in  finding  values  for  Zc>  and  the  least-squares  solution 
for  Zc  is  easily  found  to  be 

ZR-1Fr,  = Z . 

K C 

But  unlike  the  factor  scores  Zc,  these  estimators  are  intercorrelated, 
because 


(5) 
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covariance  matrix  of  the  least- squares  estimator  is  not  diagonal.  Ad- 
justing the  covariance  matrix  so  that  we  have  an  inter  cor  relation  matrix 
gives: 
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Dg  = r x r diagonal  matrix,  formed  from  the  square  roots  of  dia- 
gonal entries  of  (5),  the  covariance  matrix. 

A further  practical  difficulty  is  that  regression  estimators  are  not 
univocal,  i.  e.  , each  estimator  is  correlated  with  more  than  one  common 
factor.  This  can  be  seen  by  examining  the  matrix  of  correlations  be- 
tween the  factor  scores  and  the  least- squares  estimators,  which  is 

RfJL  = De  RL‘ 


It  is  not  too  difficult  to  derive  transformations  of  the  beta  weights 
used  in  the  regression  estimators  that  remove  these  defects,  but  it  is 
not  possible  by  means  of  a single  transformation  to  simultaneously  re- 
move both  defects. 

To  adjust  for  non-orthogonality,  we  first  find  an  arbitrary  set  of 
orthogonal  vectors  that  serves  as  a vector  basis  for  the  regression  esti- 
mators. To  do  this  we  find  an  r x r T,  such  that 

Rl  = TT' 

The  elements  of  T represent  the  correlations  of  the  least-squares  esti- 
mators with  the  r orthogonal  axes.  The  correlations  of  the  factors  with 
the  orthogonal  axes  are  given  by 

RfT  = DeT’ 

and  to  find  the  best  set  of  orthogonal  axes  we  must  transform  T so  that 
DgT  most  closely  approximates  the  diagonal  matrix  Dg.  In  matrix 
algebra,  we  wish  to  find  a 7\  , such  that  if 


R fT^  _ Dg  = E , then 
trace  E'E  = minimum,  and 

Tv!  A = i. 
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From  a theorem  proved  by  B.  F.  Green  (1)  we  find 


X ={T'D4T)-1^2  T*  D2  . 


Now  if  the  requirements  of  the  psychologist  are  such  that  it  is  more 
important  to  have  univocal  rather  than  orthogonal  estimators  than  a 
somewhat  different  procedure  is  required.  In  this  case  we  determine 
so  that  if 


R£tA  - De  = E , 
trace  E'E  = minimum,  and 

A;  Xi  = 1,  (i  = 1,  2,  . . .r),  where 
A . = i^  column  of  A . 

As  can  be  seen  the  orthogonality  restraint  on  A has  been  relaxed.  The 
normal  equations  for  these  conditions  are 


(8) 


(T'D2T 


X ) A = T'D2 


where  Y = r x r matrix  of  Lagrangian 
Multipliers.  The  above  equation  does  not  allow  us  to  find  a matrix 
expression  for  7\  which  does  not  also  involve  "Y  • Hence  we  pursue 
a simpler  approach  which  involves  a least -squares  solution  for  A 
followed  by  imposing  the  restraints  that  each  column  of  A have  sums 
of  squares  equal  to  unity.  With  this  approach  we  find  that 


A 


n 


= T *D,  where 


D is  the  diagonal  matrix  of  constants  needed  to  adjust  T~*  so  that  the 
column  sums  of  squares  equal  unity.  When  the  least-squares  estimators 
are  transformed  by  A jq  we  find  that  the  matrix  of  correlations  between 

Zc  and  our  transformed  estimators  is  = DgD,  which  is  quite  obviously 

a diagonal  matrix. 
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III.  AN  APPLICATION  OF  THE  TRANSFORMATIONS.  In  1954  APRO 
carried  out  a factor  analysis  of  visual  acuity  tests  administered  during 
dark  adaption.  The  examinees  in  this  experiment  were  100  soldiers 
from  Fort  Myer,  Virginia.  The  design  of  the  experiment  called  for  pre- 
adapting  subjects  to  a high  brightness  level  and  then  during  the  period  of 
dark  adaption  the  examinees  were  tested  on  visual  acuity  targets  pre- 
sented at  scotopic,  mesopic,  and  low  photo  pic.  brightness.  The  various 
acuity  tests  (Modified  Landolt  Ring  and  Chevron  Contrast  Adoption  tests) 
were  presented  in  a modified  Armed  Forces  Vision  Test. 


The  35  variable  intercorrelation  matrix  obtained  in  this  experiment 
yielded  8 orthogonal  factors  which  were  rotated  to  orthogonal  simple 
structure.  To  illustrate  the  derivations  presented  in  this  paper,  a sub- 
matrix was  selected  from  the  35  x 8 complete  factor  matrix.  Table  1 gives 
this  submatrix. 

Table  1 

Illustrative  Factor  Matrix 


Variable  Name 

I 

II 

III 

IV 

Landolt  Scotopic 

1 

. 84 

. 07 

. 13 

. 34 

Landolt  Low  Fhotopic 

2 

. 16 

. 85 

. 03 

. 13 

Chevron  Scotopic 

3 

. 28 

. 32 

. 12 

. 75 

Chevron  Mesopic 

4 

.01 

.03 

.79 

. 29 

Factor  I was  interpreted  to  represent  "Rod-Adapted  Resolution";  Factor  II 
was  interpreted  to  represent  "Cone  Adapted  Resolution";  Factor  III  was  in- 
terpreted to  represent  "Cone  Adapted  Brightness  Discrimination",  and 
Factor  IV  was  interpreted  as  "Rod-Adapted  Brightness  Discrimination". 


Table  2 compares  the  classic  regression  equations  approach  to  esti- 
mating these  four  factors  with  the  methods  derived  in  this  paper.  As 
noted  previously  the  least-squares  estimators  are  intercorrelated  and 
fail  to  be  univocal.  The  univocal  estimators  achieve  an  ideal  pattern  of 
correlations  with  the  factors  but  they  are  intercorrelated  somewhat  more 
than  the  least-squares  estimators.  The  orthogonal  estimators,  while  un- 
corcelated,  are  not  univocal  although  they -are  closer  to  being  univocal 
than  the  least-squares  estimators.  Inevitably  then,  choice  of  any  one 
solution  means  that  certain  defects  must  be  tolerated.  From  the  results 
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presented  in  Table  2 it  would  appear  that  the  orthogonal  estimators  re- 
present something  of  a compromise  between  the  maximum  validity  of  the 
least-squares  estimators  and  purity  of  the  univocal  estimators. 


0>~  n tt 

Green,  B.  F. 

factor  analysis. 


The  orthogonal  approximation  of  an  oblique  structure  in 
Psychometrika,  1952,  17  429-440. 


